Add 0082 OTel error-span rendering and metric#230
Merged
Conversation
PR B of proposal 0082 (the OTel rendering + the token metric; the Langfuse rendering is the final PR of the split). The per-attempt LlmRetryAttemptEvent now carries the response-side surface on a structured_output_invalid failed attempt (mirroring PR A's terminal event), so the bundled OTel observer renders it on the error openarmature.llm.complete span: a structured_output_invalid failure falls through the failed-attempt early-return to render finish_reason, usage, gated output.content, and the response identity while keeping ERROR status and category. The opt-in token-usage histogram records the failure's usage (it already recorded whenever usage was present; a structured failure now carries it). Also records the OTel exception event on every failed llm.complete span (exception.type / exception.message). The node and invocation spans already did this; the LLM span did not, a gap the 0082 error-span fixtures surfaced. This makes the enriched error_message locator load-bearing. Un-defers observability fixtures 124 (OTel error span, payload on/off) and 125 (token metric); only the Langfuse fixture 123 stays deferred.
There was a problem hiding this comment.
Pull request overview
This PR advances proposal 0082 observability by ensuring a structured_output_invalid failed attempt carries enough response-side context to be rendered on the OTel openarmature.llm.complete error span (while preserving payload redaction behavior), and by un-deferring the corresponding conformance fixtures, including the §11 token-usage-on-failure metric fixture.
Changes:
- Extend
LlmRetryAttemptEvent/OpenAI retry-attempt event construction sostructured_output_invalidfailures include response-side fields (finish_reason/usage/response id+model, plus payload-gated output content). - Update the OTel observer to (a) record an OTel semconv
exceptionevent on failed LLM attempts and (b) render the response-side surface onto the error span forstructured_output_invalid. - Un-defer and strengthen observability conformance fixture coverage for the OTel error-span rendering (124) and token-usage-on-failure metric (125), plus add focused OTel unit tests.
Reviewed changes
Copilot reviewed 8 out of 8 changed files in this pull request and generated 1 comment.
Show a summary per file
| File | Description |
|---|---|
| tests/unit/test_observability_otel.py | Adds unit tests asserting error-span surface rendering and payload redaction for structured-output failures. |
| tests/conformance/test_observability.py | Un-defers fixtures 124/125 and adds a dedicated runner + extra assertions for exception recording and payload redaction claims. |
| src/openarmature/observability/otel/observer.py | Records exception events on failed LLM attempts and renders response-side surface for structured_output_invalid error spans. |
| src/openarmature/llm/providers/openai.py | Includes response-side surface on LlmRetryAttemptEvent for StructuredOutputInvalid failures and enriches error_message. |
| src/openarmature/graph/events.py | Updates LlmRetryAttemptEvent documentation to describe the structured-output failure exception case. |
| docs/concepts/observability.md | Documents the structured-output failure exception for response-side surface rendering on error spans. |
| conformance.toml | Updates proposal 0082 note to reflect PR B’s OTel rendering + metric coverage. |
| CHANGELOG.md | Documents the new OTel error-span surface behavior and the added exception event on failed LLM attempts. |
This was referenced Jul 23, 2026
chris-colinsky
added a commit
that referenced
this pull request
Jul 23, 2026
Address CoPilot review on #230 (the guard landed after the squash-merge). _assert_error_span_extras picked candidates[0] for a span name; with a single-attempt fixture (124) names are unique, but a future fixture with multiple same-named spans (e.g. call-level retry) could silently assert the wrong span. Assert exactly one candidate so that case fails loudly here instead, rather than duplicating _assert_span_tree_matches's full name+attribute disambiguation. Test-only; no behavior change.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Phase A Wave 2 of the v0.17.0 cycle. PR B of the 0082 split (structured-output failure diagnostics, spec v0.77.0): the OTel error-span rendering + the §11 token-usage-on-failure metric. PR A (merged, #229) landed the event + error surface; PR C (Langfuse failed-Generation rendering) is the last.
What
LlmRetryAttemptEventnow carries the response-side surface on astructured_output_invalidfailed attempt (mirroring PR A's terminal event).openarmature.llm.completespan: astructured_output_invalidfailure falls through the failed-attempt early-return to renderfinish_reason,usage, gatedoutput.content, and the response identity, while keepingERRORstatus + category. Every other failure category still ends there with no response.openarmature.gen_ai.client.token.usagemetric records the failure's usage (it already recorded whenever a usage record was present; a structured failure now carries one). No metric logic change.Also fixed (observability §4.2)
The failed
openarmature.llm.completespan carriedERRORstatus but no exception event. The node and invocation spans already recorded it viarecord_exception; the LLM span did not, a gap the 0082 error-span fixtures surfaced. It now records the OTel semconv exception event (exception.type/exception.message) on every failed attempt, matching the sibling spans, which also makes the enrichederror_messagelocator load-bearing. See the CHANGELOGFixedentry.Conformance / tests
conformance.tomlkeeps 0082partial(only Langfuse remains).Reviewed with a multi-agent adversarial pass. It caught a real blocker (the missing §4.2 exception event) that my initial span-tree runner was too weak to detect; the runner is strengthened to enforce
status_description/attributes_absent/exception_recorded, and the observer records the event. Two stale-docstring findings folded in. Full suite green.